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STORE INPUT DATA 



TRACK FEATURES AND 
SELECT KEYFRAMES FROM 
INPUT FRAMES 



SELECT "TRIPLES" OF 
KEYFRAMES 





CALCULATE AND TEST 
RELATIVE CAMERA 
TRANSFORMATIONS FOR 
EACH RESPECTIVE TRIPLE 
OF KEYFRAMES TO FORM A 
SET OF KEYFRAMES FROM 
EACH RESPECTIVE TRIPLE 




1 


r 



FOR EACH TRIPLE OF 
KEYFRAMES WITH 
ADDITIONAL KEYFRAMES 
BETWEEN THE KEYFRAMES 
OF THE TRIPLE, CALCULATE 
RELATIVE CAMERA 
TRANSFORMATIONS FOR 
THE ADDITIONAL 
KEYFRAMES, AND 
REFINE THE CAMERA 
TRANSFORMATIONS FOR 
THE SET 



JOIN THE EXISTING SETS OF 
KEYFRAMES 



OUTPUT DATA 



• 



* 
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READ THE NUMBER OF INPUT FRAMES 

AND SET THRESHOLD VALUE FOR 
REDUCING THE NUMBER OF FRAMES 
TO BE PROCESSED 




IDENTIFY FEATURES IN THE FIRST 
INPUT FRAME TO BE TRACKED 



S22 



PERFORM TRACKING OF THE 
FEATURES FOR THE NEXT FRAME 




CALCULATE AN UNCERTAINTY SCORE 
FOR THE TRACKED FEATURES IN THE 
FRAME 



S26 




IDENTIFY FURTHER FEATURES IN THE 
NEW KEYFRAME TO BE TRACKED 



S32 



® 



Fig 
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® 



PERFORM TRACKING OF THE 
FEATURES FOR THE NEXT FRAME 



S34 



S36 

IS THE 
FRAME SEPARATED 
FROM THE PRECEDING 
^KEYFRAME BY MORE THAN THE. 
THRESHOLD NUMBER 
OF FRAMES? 



NO 



YES 



CALCULATE AN UNCERTAINTY SCORE 
FOR THE TRACKED FEATURES IN THE 
FRAME 



S38 



IS THE 

UNCERTAINTY SCORE LESS^ 
THAN A THRESHOLD 
VALUE? 



S40 



NO 



YES 



SET FRAME AS A KEYFRAME 



S42 



REFINE THE POSITIONS OF THE 
TRACKED FEATURES IN THE NEW 
KEYFRAME 



S44 



® 



® ® 



Fig. 5 (con 




Fig. 5 (con 
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READ CURRENT PARAMETERS OF THE 
FEATURE 



S70 



PREDICT THE VALUE OF THE 4D STATE 
AND STATE COVARIANCE MATRIX OF 
THE FEATURE FOR THE NEW FRAME 



S72 



YES 







r 




DISCARD 




£8 


FEATURE SO 






THAT IT IS NO 


S76 




LONGER 






TRACKED 





IS THE 
PREDICTED 
POSITIONAL UNCERTAINTY 
GREATER THAN A 
/THRESHOLD?^ 



S74 



SEARCH A WINDOW IN THE NEW 
FRAME TO IDENTIFY PIXELS WHICH 
MATCH THE FEATURE 



-S78 




UPDATE THE 4D STATE 
OF THE FEATURE, THE 

STATE COVARIANCE 
MATRIX AND THE 

KALMAN FILTER GAIN 



S82 



UPDATE THE 4D STATE 
OF THE FEATURE, THE 
STATE COVARIANCE 

MATRIX AND THE 
KALMAN FILTER GAIN 
USING THE IDENTIFIED 
MATCH 



Fig. 6 
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READ THE POSITIONS OF THE NEXT 
TRACKED FEATURE IN THE NEW 
KEYFRAME AND THE PRECEDING 
KEYFRAME 



S100 



DEFINE A WINDOW AROUND THE 
FEATURE IN EACH KEYFRAME 



S102 



PERFORM ADAPTIVE LEAST SQUARES 
CORRELATION OF THE PIXELS WITHIN 
THE WINDOW OF THE NEW KEYFRAME 
WITH THE PIXELS WITHIN THE 
WINDOW OF THE PRECEDING 
KEYFRAME 



DISCARD THE 
MEASUREMENTS FOR 
THE FEATURE IN THE 
NEW KEYFRAME AND 
UPDATE THE 4D 
STATE OF THE 
FEATURE, THE STATE 
COVARIANCE MATRIX 
AND THE KALMAN 
FILTER GAIN 



S108 



S106 




UPDATE THE 4D 
STATE OF THE 
FEATURE, THE STATE 
COVARIANCE MATRIX 
AND THE KALMAN 
FILTER GAIN USING 

THE REFINED 
POSITION OF THE 
FEATURE 



S110 
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SET THE FIRST KEYFRAME IN THE 
SEQUENCE AS FRAME 1 OF THE FIRST 
TRIPLE 



S132 



CONSIDER THE NEXT KEYFRAME IN 
THE SEQUENCE 



S134 



NO 



IS THE 
NUMBER OF TRACKED^ 
FEATURES WHICH ARE IN 
BOTH THE CURRENT KEYFRAME 
AND FRAME 1 OF THE 
TRIPLE LESS THAN A 
THRESHOLD? 



S136 



S140 



REMOVE FRAME 1 FROM 
THE TRIPLE AND SET 
THE CURRENT 
KEYFRAME AS FRAME 1 
OF A NEW TRIPLE 




YES 



S138 



NO 



IS THERE 
"AT LEAST ONE KEYFRAME* 
BETWEEN THE CURRENT 
KEYFRAME AND FRAME 1 
OF THE TRIPLE? 



YES 



SET THE KEYFRAME PRECEDING THE 
CURRENT KEYFRAME AS FRAME 3 OF 
THE TRIPLE 



S142 



REMOVE 
FRAME 3 FROM 
THE TRIPLE 



S146 



© 



Fig. 9 



© 
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S144 




S152 



SET THE MIDDLE 
KEYFRAME BETWEEN 
FRAME 1 AND FRAME 3 AS 
FRAME 2 OF THE TRIPLE 



SET FRAME 2 OF THE TRIPLE TO 
BE THE FRAME FROM THE 
MIDDLE PAIR OF KEYFRAMES 
WHICH CONTAINS THE HIGHEST 
NUMBER OF TRACKED 
FEATURES WHICH ARE ALSO 
PRESENT IN FRAME 1 AND 
FRAME 3 OF THE TRIPLE 



J2_ 



CONSIDER NEXT KEYFRAME OF THE 
KEYFRAMES AFTER FRAME 3 OF THE 
PRECEDING TRIPLE 



S154 



NO 



IS THE 
NUMBER OF TRACKED 
FEATURES WHICH ARE IN 
BOTH THE CURRENT KEYFRAME 
\ND FRAME 3 OF THE PRECEDING 
TRIPLE LESS THAN A 
THRESHOLD? 



S156 



YES 



SET THE KEYFRAME AS FRAME 1 OF 
A NEW TRIPLE 



S158 



Fig. 9 (cont) 
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READ DATA FOR NEXT TRIPLE 
OF KEYFRAMES 



S180 



CALCULATE AND TEST RELATIVE 
CAMERA TRANSFORMATIONS 
FOR THE TRIPLE 



S182 





Fig. 10 
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READ DATA FOR NEXT REMAINING 
TRIPLE OF KEYFRAMES 



S200 




CALCULATE RELATIVE CAMERA 
TRANSFORMATIONS FOR EACH 
KEYFRAME LYING BETWEEN 
KEYFRAMES OF THE TRIPLE 



S204 



ADJUST THE CAMERA 
TRANSFORMATIONS FOR THE SET 
OF KEYFRAMES COMPRISING THE 
TRIPLE OF KEYFRAMES AND THE 
KEYFRAMES THEREBETWEEN 



S206 




Fig. 12 
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t 

CALCULATE A FEATURE POINT IN 3D 
FOR EACH TRACKED FEATURE WHICH 
IS PRESENT IN AT LEAST TWO OF THE 
THREE KEYFRAMES OF THE TRIPLE , 



READ DATA FOR THE NEXT KEYFRAME 
WHICH IS BETWEEN THE KEYFRAMES 
OF THE TRIPLE 



LIST CORRESPONDENCES BETWEEN 
THE 3D FEATURE POINTS AND THE 
POSITIONS OF THE TRACKED 
FEATURES IN THE INTRA-TRIPLE 
KEYFRAME 



SET COUNTER = COUNTER + 1 



SELECT 3 CORRESPONDENCES AT 
RANDOM 



CALCULATE CAMERA 
TRANSFORMATIONS FOR THE INTRA- 
TRIPLE KEYFRAME USING THE 
SELECTED CORRESPONDENCES 



TEST THE CALCULATED 
TRANSFORMATIONS AGAINST EACH 
CORRESPONDENCE 



S220 



S222 



S224 



S226 



S228 



S230 



S232 



Fig 
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ARE THE 
TRANSFORMATIONS 
MORE ACCURATE THAN ANY 
PREVIOUSLY CALCULATED 
^FOR THE INTRA-TRIPLE. 
KEYFRAME? 



S234 



NO 



YES 



STORE THE CALCULATED 
TRANSFORMATIONS AND ASSOCIATED 
ACCURACY 



S236 




® 



YES 
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DISCARD THE INTRA-TRIPLE 
KEYFRAME 



S244 




S246 



NO 



Fig. 13 (con 



FEATURE POINT 1 



FEATURE POINT 2 



FEATURE POINT 3 



FEATURE POINT n 





y 2 



260 



Y 

3D POSITION 



262 



J 



2D POSITION IN 
INTRA-TRIPLE 
KEYFRAME 



Fig. 14 



• # 
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Fig. 15c 
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READ THE 2D POSITION OF EACH 
TRACKED FEATURE IN EACH OF THE 
KEYFRAMES WITHIN THE NEXT 
WINDOW 



S270 



CALCULATE AN ERROR 
MEASUREMENT VALUE FOR THE 
SUBSET OF KEYFRAMES CURRENTLY 
BEING CONSIDERED USING THE 
CALCULATED CAMERA 
TRANSFORMATIONS AND THE 
POSITIONS OF THE TRACKED 
FEATURES WITHIN THE KEYFRAMES 
AND IN 3D 



ADJUST THE CAMERA 
TRANSFORMATIONS AND THE 
POSITIONS OF THE TRACKED 
FEATURES IN 3D TO MINIMISE THE 
ERROR MEASUREMENT VALUE 



S274 
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CALCULATE CAMERA 
TRANSFORMATIONS FOR 
KEYFRAMES BETWEEN SETS BY 
EXTENDING THE SETS OF 
KEYFRAMES 



S290 



MERGE SETS OF KEYFRAMES 
WHICH MEET 



S292 



MERGE SETS OF KEYFRAMES 
WHICH DO NOT MEET 



S294 



v 



Fig. 19 
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DETERMINE A PROCESSING ORDER 
FOR THE ENDS OF THE SETS OF 
KEYFRAMES 



CONSIDER NEXT END KEYFRAME 



CONSIDER THE KEYFRAME IN THE 
GAP ADJACENT TO THE END 
KEYFRAME 



S300 



S302 



S304 



READ THE TRACKED FEATURES IN THE 
GAP KEYFRAME AND IDENTIFY 
CORRESPONDING 3D POINTS. IF 3D 

POINT DOES NOT EXIST FOR A 
FEATURE AND THE FEATURE IS ALSO 

PRESENT IN AT LEAST TWO 
KEYFRAMES OF THE SET, CALCULATE 
3D POINT 



S306 



CALCULATE RELATIVE CAMERA 
TRANSFORMATIONS FOR THE 
KEYFRAME 



S308 




DISCARD THE 
CALCULATED 
CAMERA 
TRANSFORMATION 
AND DELETE THE 

CURRENT END 
KEYFRAME FROM 
THE PROCESSING 
ORDER 



S314 



S312 



->T<~ 



CURRENT KEYFRAME 

BECOMES PART OF 
THE SET. ADJUST THE 
CALCULATED CAMERA 
TRANSFORMATIONS 
FOR THE SET 




Fig. 20 
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NO 



Fig. 20 (cont) 



CONSIDER NEXT PAIR OF KEYFRAME 
SETS WHICH MEET 



S330 



REGISTER THE CALCULATED 
CAMERA TRANSFORMATIONS FOR 
THE KEYFRAMES ADJACENT THE 
BOUNDARY 



S332 




Fig. 22 
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# 



CONSIDER ALL KEYFRAMES IN THE 
FIRST SET AND ALL KEYFRAMES IN 
THE SECOND SET 




f 


IDENTIFY EACH TRACKED FEATURE 
WHICH IS IN AT LEAST TWO 
KEYFRAMES FROM EACH SET 




r 



READ THE 3D POSITION OF EACH 
FEATURE POINT CORRESPONDING TO 
AN IDENTIFIED FEATURE WHICH HAS 

PREVIOUSLY BEEN CALCULATED 
USING THE KEYFRAMES IN THE FIRST 
. SET 



READ THE 3D POSITION OF EACH 
FEATURE POINT CORRESPONDING TO 
AN IDENTIFIED FEATURE WHICH HAS 
PREVIOUSLY BEEN CALCULATED 
USING THE KEYFRAMES IN THE 
SECOND SET 



USE THE 3D POSITIONS OF 
CORRESPONDING FEATURE POINTS 
TO CALCULATE THE 
TRANSFORMATION BETWEEN THE 
COORDINATE SYSTEMS OF THE FIRST 
AND SECOND KEYFRAME SETS 



Fig. 23 



• 
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CONSIDER NEXT GAP BETWEEN 
CONSECUTIVE PAIRS OF KEYFRAME 
SETS 



S370 



DISCARD THE KEYFRAMES WITHIN 
THE GAP 



S372 



MATCH FEATURES IN THE KEYFRAMES 
ADJACENT THE GAP 



S374 



IS NUMBER 
YES OF MATCHED FEATURES 

LESS THAN A 
THRESHOLD? 



S376 




USE THE KEYFRAMES IN THE SET 
CONTAINING THE FIRST KEYFRAME 
ADJACENT THE GAP TO CALCULATE A 
FEATURE POINT IN 3D FOR EACH 
FEATURE MATCHED IN THE 
KEYFRAMES ADJACENT THE GAP FOR 
WHICH A FEATURE POINT HAS NOT 
ALREADY BEEN CALCULATED 



S378 



USE THE KEYFRAMES IN THE SET 
CONTAINING THE SECOND KEYFRAME 
ADJACENT THE GAP TO CALCULATE A 
FEATURE POINT IN 3D FOR EACH 
FEATURE MATCHED IN THE 
KEYFRAMES ADJACENT THE GAP FOR 
WHICH A FEATURE POINT HAS NOT 
ALREADY BEEN CALCULATED 



® 



® 



Fig. 25 



# 



4 
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USE THE MATCHED FEATURES TO 
REGISTER THE CAMERA 
TRANSFORMATIONS FOR THE 
KEYFRAMES ADJACENT THE GAP 



S382 




NO 



Fig. 25 (con 



